As a display apparatus which can display a three-dimensional image (stereoscopic image), various systems are known. In recent years, especially, a system, which adopts a flat-panel type, and displays a stereoscopic image without requiring any dedicated glasses and the like, is demanded more strongly. There has been developed a system which is provided with a display panel (display device) and a parallax barrier (also called a ray control element) arranged in front of a display screen of the display panel (display device). The display panel displays an image or a picture on the display screen having pixels fixed on a plane, like a direct-view or projection type stereoscopic moving image display device (for example, a liquid crystal display device or plasma display device) and the parallax barrier controls rays coming from the display panel to direct them toward an observer. This system is a practical system which can relatively easily display a stereoscopic image.
A so-called parallax barrier controls rays so as to allow an observer to observe different images depending on observation angles even when the observer observes the same parallax barrier position. More specifically, when a right-and-left parallax (horizontal parallax) is given, slits or a lenticular sheet (cylindrical lens array) is used as the parallax barrier. When both the right-and-left parallax and an up-and-down parallax (vertical parallax) are given, a pinhole array or lens array is used as the parallax barrier. In this specification, one slit or one lens as a unit of the parallax barrier is called an exit pupil.
The system using the parallax barrier is further classified into a two-view system, multi-view system, ultra-multi-view system (ultra-multi-view conditions are given to the multi-view system), and integral imaging (to be simply referred to as “II” hereinafter) system. The basic principle of these systems is substantially the same as a stereoscopic photo system invented about 100 years ago. However, since the number of pixels of a display device is finite, the number of pixels assigned per exit pupil is also finite. In this specification, the number of pixels assigned per exit pupil is called the number of parallaxes, and a two-dimensional image configured by pixels assigned to respective exit pupils is called an element image.
Note that the II system is a term of stereoscopic photography, and is also called integral photography (to be also abbreviated as IP hereinafter).
In order to display a stereoscopic image using these II systems, images (multi-viewpoint images) captured from a plurality of directions are required. That is, in a stereoscopic image display method based on the two-view system, two multi-viewpoint images are prepared. In a stereoscopic image display method based on the multi-view system or II system, multi-viewpoint images as many as the number of pixels corresponding to the number of parallaxes assigned per exit pupil are prepared. In this specification, a pixel means a minimum display unit. Basically, multi-viewpoint images are captured under the precondition of the relationship between pixels and exit pupils. A multi-viewpoint image generation method includes a plurality of generation methods such as actual imaging and CG rendering. However, multi-viewpoint images are normally prepared by actual imaging that captures images of an object using cameras.
In the actual imaging using the cameras, more specifically, cameras as many as the number of parallaxes, which are used to capture multi-viewpoint images, are laid out, so as to be symmetrical to a relationship between exit pupils and corresponding pixel positions. The cameras laid out to capture multi-viewpoint images are called a multi-camera. Since pixels of a display device are arranged on a plane, the multi-camera is similarly arranged on a plane. In a stereoscopic display device, letting pp be a pixel interval, and g be an interval between an exit pupil and a pixel plane of the display device, an imaging reference distance Lc and interval x_c of a multi-camera 1 in the stereoscopic display device are given by:g:pp=Lc:x_c
This imaging condition means that it is most efficient to match a size and resolution of an imaging reference plane of the multi-camera with those of a flat-panel display unit in the display device, so as to satisfy the imaging condition in the multi-camera in a stereoscopic imaging device and the flat-panel display unit in the stereoscopic display device. In this case, the imaging reference plane is called a projection plane under the precondition that it is matched with the display screen, the imaging reference distance is set as an observation reference visual distance of a three-dimensional display, and an imaging position is set as a viewpoint on the observation reference plane of the three-dimensional display. In addition, rays at the time of imaging and playback agree with each other, and an image of an object to be captured is displayed in a real scale.
However, this actual imaging condition need not always be strictly satisfied. In recent years, when it is designed to observe information of neighboring pixels to be mixed to some extent, it is devised to allow an observer to observe a stereoscopic image even outside an observation distance range as disclosed in R. Fukushima et al., Proceedings of SPIE-IS & T Electronic Imaging, 7237, 72370W-1 (2009). Furthermore, in a three-dimensional display based on the parallax barrier system, a display range in its z direction (a direction perpendicular to the display screen) is limited as disclosed in J. Opt. Soc. Am. A vol. 15, p. 2059 (1998). Therefore, a multi-camera which captures multi-viewpoint images more than the number of viewpoints is prepared, multi-viewpoint images having an interval x_c smaller than a design value are selected from the multi-camera, and images which are compressed in the z direction are often displayed as disclosed in JP-A 2005-331844 (KOKAI). In this case, the z direction means a depth direction which is perpendicular to a horizontal direction x and vertical direction y of a three-dimensional display screen, and corresponds to a back surface side of the display screen. Also, a method of displaying a stereoscopic image within a display range by shifting z coordinates of existing multi-viewpoint images upon displaying the stereoscopic image and enlarging or reducing them in the x and y directions, so as to adjust clipping ranges used as parallax images, that is, clipping methods is known as disclosed in JP-A 2004-343290 (KOKAI). These literatures merely disclose a display method of a stereoscopic image to be displayed by selecting already captured multi-viewpoint images or adjusting clipping ranges.
In order to change z coordinates upon displaying a stereoscopic image, more specifically, a shift value for each viewpoint image within a range used as a parallax image need only be changed. However, in case of actual imaging, since multi-viewpoint images are perspective projection images, when the projection plane is shifted forward or backward along the z axis upon changing the shift value, the imaging reference distance is different from the observation reference visual distance of the three-dimensional display, and a distortion is generated in a strict sense. In order to display a stereoscopic image free from any distortion, the imaging reference distance to an object to be mainly displayed has to be set to be equal to the observation reference visual distance of the three-dimensional display in place of the acquired multi-viewpoint images which have undergone post-processing, and are reconstructed to display a stereoscopic image. However, there is no method which allows a photographer to correctly recognize the imaging reference distance, and the imaging reference distance cannot be correctly set. Also, there is no method which allows a photographer to know which object an observer of the three-dimensional display located at a remote place wants to mainly and stereoscopically display. Since this object to be stereoscopically displayed does not become clear, the imaging reference distance cannot be set due to that cause.